Skip to content

fix(nightly): refresh the App token across stages; a lost watch is a driver error - #578

Merged
Jammy2211 merged 2 commits into
mainfrom
feature/nightly-token-refresh
Nov 4, 2026
Merged

Jammy2211 merged 2 commits into
mainfrom
feature/nightly-token-refresh

Conversation

@Jammy2211

Copy link
Copy Markdown
Contributor

Summary

The nightly release driver runs for hours on one GitHub App installation token that lives 60 minutes (since 6f5f95e, 2026-10-23). Every scheduled night from 10-24 to 11-04 died at the 60-minute mark with HTTP 401: Bad credentials while watching the Stage 3 Heart integrate run, and paged "Stage 3 failed" although that run passed on most nights. No nightly reached the readiness gate after the 2026-10-22 anchor; the 11-01 release was manual. NIGHTLY_RELEASES is false since 2026-11-04 until this lands.

This obeys the token policy as written ("mint it right before a long loop, not at the top of the job") with no PAT and the same four permissions:

  • bin/gh_app_token.py — stdlib + openssl mint of a fresh installation token; prints only the token, never logs the key or JWT.
  • agents/conductors/release/_nightly_lib.sh — refresh_gh_token re-mints once the token is 45 min old (no-op without App credentials, so local runs on a personal gh are unchanged); await_run polls a run every minute, refreshing as it goes, and returns 4 after five consecutive failed reads.
  • agents/conductors/release/nightly.sh — refresh before every dispatch, download, read and the anchor write; a watch that still loses its credentials pages as a driver error (exit 1, workflow outcome driver-error), never as a stage failure. Step 6 uses notify rather than page on that path because the live release may have published.
  • .github/workflows/nightly-release.yml — the driver step receives APP_CLIENT_ID / APP_PRIVATE_KEY for re-minting only; the initial mint step and the OUTCOME CONTRACT are unchanged.

Fixes #577.

API Changes

None. No library or public Python API is touched; the driver's exit-code contract gains one documented meaning for exit 1 (a lost watch). dispatch_and_await gains return code 4 (internal to the driver).

Test Plan

  • python3 -m pytest -q tests/test_gh_app_token.py tests/test_nightly_await.py tests/test_activity_gate.py — 23 passed
  • python3 -m pytest -q -x (whole Brain suite) — 1558 passed, 1 skipped
  • bash -n on the driver and the new lib
  • Witness (after merge, before re-arming): one manual nightly-release.yml dispatch with dry_run=true whose log shows Stage 2 and Stage 3 complete, no HTTP 401, and step 5 — readiness gate reached (Heart will likely be STALE, so it ends at exit 3 — that is the gate, not the bug). Then set NIGHTLY_RELEASES=true.

Decisions flagged for review

  • The token request names the Variables permission actions_variables, matching the installation's own name (e0f88ca, 2026-10-23) and what create-github-app-token sends. If GitHub rejects it (HTTP 422 in the mint's stderr), the dry-run Witness will show it; the constant and the parity test change together.
  • The App private key now enters the driver step's environment. The same job already holds it for the mint action; the driver is deterministic shell with no tracing and never echoes its environment.
Full API Changes (for automation & release notes)

Added

  • bin/gh_app_token.py — make_jwt(client_id, private_key, now=None), mint(client_id, private_key, owner="PyAutoLabs", request=_request), INSTALLATION_PERMISSIONS
  • agents/conductors/release/_nightly_lib.sh — refresh_gh_token [--force], await_run <repo> <run_id>; env overrides NIGHTLY_POLL_SECONDS, NIGHTLY_MINT_CMD (tests)

Removed

  • the single blocking gh run watch in dispatch_and_await

Generated by the PyAutoLabs agent workflow.

🤖 Generated with Claude Code

…driver error

The nightly driver ran a 1.5-4.5 h job on one App installation token that
lives 60 minutes (since 6f5f95e, 2026-10-23). Every scheduled night from
10-24 to 11-04 died at the 60-minute mark with 'HTTP 401: Bad credentials'
inside the Stage 3 gh run watch and paged 'Stage 3 failed' over a run that
had passed.

- bin/gh_app_token.py: stdlib + openssl mint of a fresh installation token
  with the workflow's exact permission set (actions write, contents read,
  issues read, actions_variables write).
- agents/conductors/release/_nightly_lib.sh: refresh_gh_token re-mints once
  the token is 45 min old; await_run polls a run (refreshing as it goes) and
  returns 4 when the run cannot be observed after five failed reads.
- nightly.sh: refresh before every dispatch, download, read and the anchor
  write; a lost watch pages as a driver error (exit 1), never a stage failure.
- nightly-release.yml: the driver step receives APP_CLIENT_ID/APP_PRIVATE_KEY
  for re-minting only; the initial mint step and the outcome contract stand.

Closes #577

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The tenant firewall forbids a new organ file from naming the GitHub owner.
The mint helper now requires APP_OWNER (set in nightly-release.yml beside the
mint step's own owner:), and the test uses a neutral owner.

Refs #577

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Jammy2211
Jammy2211 merged commit 61a3f87 into main Nov 4, 2026
2 checks passed
@Jammy2211
Jammy2211 deleted the feature/nightly-token-refresh branch November 4, 2026 18:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(nightly): refresh the App token across stages; a lost watch is a driver error

1 participant